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Executive Summary 


Some students, including students with print disabilities, students with low vision, and English lan- 
guage learners, may benefit from having test content read aloud. However, there have been challenges 
in standardizing the presentation of test content, including whether or not to read answer choices or 
to describe maps and cartoons, among many other issues. Technology-enhanced assessment offers 
the opportunity to provide standardized audio support delivered through the test platform. One goal 
of the Guidelines for Accessible Assessment Project (GAAP) was to develop a set of audio guidelines 
that could be used across states. 

This report features the findings from cognitive labs conducted with students to gain their input and 
feedback on the proposed guidelines. A total of 46 students participated in the audio cognitive labs, 
including 22 students with print disabilities, 17 students with low vision, and seven students who are 
English language learners. 

Key findings include: 

• Eindings were generally mixed, but some patterns emerged. Overall, students reported that audio 
support helped them understand test questions better. Students also typically preferred to have 
numbers read aloud in both the answer stem and answer choices. In several of the item sets, one 
item in the pair would have numbers read aloud, and in the other item, numbers would be read 
as “this number” or “equation.” In all of these cases, students in all three demographic groups 
preferred the item where the numbers were included in the audio support. 

• Students also had a slight preference for numbers with decimals to be read as numerals with 
“decimal point” rather than to be read with place value. 

• Students were mixed in their preferences for audio support with the drag-and-drop. Students 
reported a range of experiences with this item type. Eor some students, this was a new item type. 
Other students were familiar with the drag-and-drop feature from computer games or instruction 
but had not encountered it in an assessment. A few students were familiar with this item type. 

Overall, the findings from the cognitive labs support the need for taking an individualized approach in 
providing audio support for assessments. In developing online assessments, it is important to program 
audio support in ways that allow for multiple- approach (e.g., all parts of the item, a stimulus only, 
answer choices only, etc.) read aloud. 
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Introduction 


The Elementary and Secondary Education Act (ESEA) and the Individuals with Disabilities 
Education Improvement Act (IDEA) both require that accommodations be provided, as appropri- 
ate, to those students with disabilities who need them during assessments. If students’ individual 
accessibility and accommodations needs are not addressed appropriately, such students are placed 
at a severe disadvantage in demonstrating what they know and can do on state assessments. 

Many states and assessment consortia such as the Partnership for Assessment of Readiness for 
College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (Smarter 
Balanced) have prepared guidelines regarding the roles and responsibilities of individuals 
who assist in the administration of accessibility features and accommodations. However, these 
guidelines often require additional detail to provide valid and equitable support for students who 
need audio-based accessibility features and accommodations. Eurthermore, individuals who 
are tasked with reading aloud test content rarely see the test content in advance due to security 
restrictions. Therefore, they need to be empowered by comprehensive guidelines, as they often 
have to read content aloud on the spot. 

The Guidelines for Accessible Assessment Project (GAAP) was implemented to meet the fol- 
lowing goals: 

1. to develop research-based audio guidelines that can be used across states, consortia, and 
assessment vendors to produce reliable and valid audio representations of assessment items 
and tasks for students with low vision or reading-based learning disabilities, and students 
who are English language learners (EEEs), and 

2. to develop research-based sign guidelines that can be used across states, consortia, and as- 
sessment vendors to produce reliable and valid signed representations of assessment items 
and tasks for students who communicate using sign language. 

The project was conducted to inform statewide assessments that are more accessible to EEEs 
and students with low vision or reading-based learning disabilities, than previous iterations. 
Data collected through student cognitive labs have implications for developing accessible tests 
for students with a variety of assessment needs. This report highlights evidence collected to 
address the first goal of the project: the development of research-based audio guidelines for 
use on statewide assessments. The following is the research question addressed in the report: 

• What audio accessibility guidelines are the most usable and most effective for students to 
use on assessments? 


NCEO 


1 


This question was addressed by gathering input from state education agency personnel, including 
assessment and content experts. However, there is very little consensus on the best approaches 
to deliver audio support without violating the construct being measured. This report specifically 
addresses an additional component to this question: gathering input and feedback from students 
themselves on how assessment content should be read aloud. 


Method 

Cognitive labs were conducted with students at elementary, middle, and high school grade 
levels. Participating students came from states affiliated with PARCC, Smarter Balanced, and 
non-consortium-affiliated states. Students were selected based on having either a reading-based 
learning disability (LD-Reading), low vision, or having been identified as an ELL. 

A total of 46 students completed the cognitive labs. As shown in Table 1, 29 students were male, 
and 17 were female. An equal number of students participated in the middle- and high-school 
(n=17) versions of the assessment, with 12 students participating in the elementary school 
version. The LD-Reading group was the largest (n=22), followed by students with low vision 
(n=17), and ELLs (n=7). Students were recruited from a total of three states for the audio sup- 
port cognitive labs. 

Students who participated in the audio support cognitive labs ranged in their experience with 
using audio support for instruction and assessment. When asked about their experiences with 
audio support, some students commented that they were unsure if they had received audio sup- 
port for testing. Other students noted that they had just started using audio support as a testing 
accommodation. Two students commented that they had received audio support from a teacher 
who had read test questions aloud. They noted that they liked the teacher reading the test out 
loud to them. 

Students were asked if they thought the audio support was helpful to them in understanding test 
questions. A total of four students stated that they thought the audio support was not helpful. 
One student did not give a response. The remaining 41 students stated that they thought the 
read aloud was helpful. Students’ comments about how it was helpful included, “it makes more 
sense when they say it for you,” “sometimes I can’t really focus on what I’m reading,” “I have 
more confidence that I know what is being asked,” and “some stuff can’t be put on braille maps 
or pictures, so it helps with understanding.” 

Each student was provided a brief test with isomorph item sets in which one feature of the item 
pair varied (e.g., in the first item, the cartoon was described, and in the second item, it was not). 
See Appendix A for the cognitive lab instruments used in this study. During the assessment. 
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two researchers observed students and took notes on their interactions with the items. The 
researchers were interested in how earnestly students were responding to test items. Students 
were observed to not answer test items earnestly if they reported that they guessed at the answer 
or if they responded to the item pair quickly, selecting the same answer choice for both items. 
In addition, the researchers noted how many times the student fully or partially replayed the 
audio for each item. 

Following the completion of each item pair, students were asked several questions about the 
isomorph items, including how difficult they thought the item was, their item preference related 
to the variation between item pairs, and what aspects of the audio presentation for the items 
were helpful or not helpful. 

Table 1 . Student characteristics for the audio assessment 


Characteristics 

LD-Reading 

ELL 

Low Vision 

Total 

Sex 





Male 

16 

4 

9 

29 

Female 

6 

3 

8 

17 

Grade 





Elementary 

5 

4 

3 

12 

Middle School 

10 

3 

4 

17 

High School 

7 

0 

10 

17 

TOTAL 

22 

7 

17 



The results of this report are laid out in three sections. The first section covers responses from 
elementary school students, followed by middle school, then high school student responses. A 
synthesis of responses by student need across grade levels is provided at the end of the three 
sections. 


Elementary School Assessment With Audio Support 

The elementary school assessment with audio support included six questions regarding student 
preferences for how items were presented, including the following: 

1. feedback on the level of detail of audio description of items, 

2. feedback on drag-and-drop items, 

3. whether mathematical expressions were read aloud, 

4. whether shapes in figures were described. 
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5. whether answer options were read, and 

6. whether students preferred numbers to be read in items. 

Each issue is discussed below. 

At the elementary school level, most students were observed to have answered the items ear- 
nestly. In one instance, a student with a reading-based learning disability answered immediately 
for both items, but was correct for both items, making it difficult to determine whether he knew 
the material or guessed correctly. In some instances, several students were observed to be tak- 
ing their time to listen to the computer directions and drop the items into the boxes to match 
the item fraction. 

Issue 1 : Do students prefer that detailed information on coordinate grids be 
read aioud or prefer that the grid oniy be described more generaiiy? 

An item pair was developed in which the first item provided detailed information about the 
coordinate grid, while the second item provided a general description of the content. 

Replaying Audio 

Only one student, a student with low vision, replayed the audio for this item pair. 

Reported Difficulty 

After each item pair, elementary school students were asked about their perceptions related to 
the difficulty level of the items. Tables 2 and 3 show the reported difficulty of items in this set 
according to the elementary school students who responded to Item Set I . Students were split 
about whether they thought the item sets were easy or difficult to complete. For students who 
responded to the question of item difficulty, half of ELLs and students with Eow Vision reported 
that the items were easy, while the other half reported that the items were difficult. 


Table 2. Student reported item difficulty for Item 1 : detailed description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

0 

0 

1 

5 

ELL (n=4) 

2 

0 

2 

0 

Low Vision (n=3) 

1 

0 

1 

1 

TOTAL(n=12) 

3 

0 

3 

6 


4 
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Table 3. Student reported item difficulty for Item 2: general description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

0 

0 

1 

5 

ELL (n=4) 

2 

0 

2 

0 

Low Vision (n=3) 

1 

0 

1 

1 

TOTAL(n=12) 

3 

0 

3 

6 


Item Preferences 


The majority of students preferred Item 1 (Table 4). However, students who had a reading-based 
learning disability preferred Item 2 (n=3) than Item 1 (n=2) about the same. For the other de- 
mographic categories, the majority of students preferred Item 1. 


Table 4. Elementary school students’ item preferences by participant group 


Student Demographic Category 

Preferred 
Item 1 
(Detailed 
Description) 

Preferred 
Item 2 
(General 
Description) 

No Preference 

LD-Reading (n=5) 

2 

3 

0 

ELL (n=4) 

3 

0 

1 

Low Vision (n=3) 

3 

0 

0 

TOTAL(n=12) 

8 

4 

0 


Students who preferred Item 1 typically stated that they liked how the computer read the loca- 
tion of each house in the coordinate grid. Students who preferred the second item either stated 
that they liked that the read aloud ended quicker than in the first item, or that they had “become 
used to” the item and used this to help them answer the second item. The student without a 
preference did not notice any difference between the two items. 

Issue 2: Do students prefer hearing audio feedback for drag-and-drop items 
(feedback teiis the student what object was moved or what the drop area iooks 
iike with dragged objects)? 

To answer this research question, an item pair was developed where students were asked to 
drop either soybeans or corn kernels into a grid to match a fraction in the item (for example, 
the first item tells students that a farmer planted three-quarters of his field with soybeans, and 
they have to drop the correct amount of soybeans into an eight-box grid to match the fraction). 
For the first item, the students were told when they had dropped a soybean in the field and how 
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many fields were planted. The second item did not provide feedback when students dropped 
com into the boxes. 

Replaying Audio 

The majority of students did not replay any of the audio for Item Set 2, but one ELL and one 
student with low vision replayed Item 3. The same student with low vision replayed Item 4. 

Reported Difficulty 

Tables 5 and 6 show the reported difficulty level for items in Set 2. All students who reported 
a difficulty level for these two items found the items to be Easy. 


Table 5. Student reported difficulty for Item 3: drag-and-drop audio feedback 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

r 

3 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

5 

0 

0 

7 


Table 6. Student reported difficulty for Item 4: drag-and-drop no audio feedback 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

0 

4 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

5 

0 

0 

7 


Item Preferences 

The majority of students preferred Item 3 to Item 4 (Table 7). However, student preference 
varied by need. Students with a reading-based learning disability were more likely to prefer 
Item 3 (n=3) than Item 4 (n=2). ELLs were evenly split on their preference for the two items. 
All students who had low vision preferred Item 3 (n=3), which tells the student when a seed is 
dropped into the box and how many seeds are in the boxes. 
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Table 7. Elementary school students’ Item preferences by participant group 


Student Demographic Category 

Preferred Item 
3 (Drag-and- 
Drop Audio 
Feedback) 

Preferred Item 
4 (Drag-and- 
Drop no Audio 
Feedback) 

No Preference 

LD-Reading (n=5) 

3 

2 

0 

ELL (n=4) 

2 

2 

0 

Low Vision (n=3) 

3 

0 

0 

TOTAL(n=12) 

8 

4 

0 


The majority of students who preferred Item 3 stated that they liked that the computer affirmed 
when they had put a seed into the drag-and-drop region, and that it told them how many seeds 
were in the grid. Students who preferred the second item reported that they thought the com- 
puter voice was “annoying” when it updated them on how many seeds had been dropped into 
the grid, and that they liked how quickly they could complete Item 4 because the computer was 
not updating them with the count of seeds each time they dragged one into the grid. 

Issue 3: Do students prefer that expressions and equations be read aioud? 

To answer this question, an item pair was developed that contained one question in which the 
expressions and equations were read aloud and one question in which the expressions and equa- 
tions were not read aloud. 

Replaying Audio 

One student with low vision replayed the audio for Item 5, where equations were read. 

Reported Difficulty 

Tables 8 and 9 show the reported difficulty levels for Item Set 3. Of the students who reported 
a difficulty level, more students thought the items were Easy. A total of three students reported 
both Item 5 and Item 6 to be Difficult. 
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Table 8. Student reported difficulty of Item 5: equations read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

1 

3 

ELL (n=4) 

2 

0 

2 

0 

Low Vision (n=3) 

1 

1 

0 

1 

TOTAL(n=12) 

4 

1 

3 

3 


Table 9. Student reported difficulty of Item 6: equations not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

1 

3 

ELL (n=4) 

2 

0 

2 

0 

Low Vision (n=3) 

2 

1 

0 

0 

TOTAL(n=12) 

5 

1 

3 

3 


Item Preferences 


In general, students preferred the first item in the pair, where the expressions and equations 
were read aloud. Students with a reading-based learning disability were somewhat even in their 
preferences for both items. Other groups tended to prefer Item 5. Table 10 shows the preference 
for each item by student need. 


Table 10. Elementary school students’ item preferences by participant group for Item Set 3 


Student Demographic Category 

Preferred Item 
5 (Equations 
Read) 

Preferred Item 6 
(Equations not 
Read) 

No 

Preference 

LD-Reading (n=5) 

2 

3 

0 

ELL (n=4) 

2 

1 

1 

Low Vision (n=3) 

3 

0 

0 

TOTAL(n=12) 

7 

4 

1 


Students who preferred Item 5 liked that the computer read the equation to them. One student 
commented that as the computer read the problem, he could think about it since they were not 
trying to read it. Another mentioned that the computer explained parts of the equation they did 
not understand. Students who preferred Item 6, where the equation was not read, stated that 
they understood the equation and did not need the computer to read it to them. 
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Issue 4: Do students prefer that diagrams be described? 


An item pair was developed to answer whether students prefer that diagrams be described to 
them. In Item Set 4, the first item described three shapes that were in a diagram, while the second 
stated that there was a diagram and three shapes, but did not name the shapes. 

Replaying Audio 

For this item set, one student in the LD-Reading group replayed the audio for Item 8, where 
the shapes were not described. 

Reported Difficulty 

Tables 1 1 and 12 show the reported difficulty levels for Item Set 4. Only about half of the stu- 
dents reported a difficulty level for this item pair. Of those who reported a difficulty level, most 
found the items to be Easy. 


Table 1 1 . Student reported difficulty of Item 7: shapes described 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

3 

0 

0 

2 

ELL (n=4) 

2 

0 

1 

1 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

5 

0 

1 

6 


Table 12. Student reported difficulty of Item 8: shapes not described 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

2 

0 

1 

2 

ELL (n=4) 

3 

0 

1 

0 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

5 

0 

2 

5 


Item Preferences 


As shown in Table 13, students were more likely to prefer Item 7, where the shapes within the 
diagram were described, than they were to prefer when the shapes were not described (Item 8). 
However, preferences varied by student need. Students with reading-based learning disabilities 
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were evenly split in item preference, or had no preference (n=l). ELLs and students with low 
vision were more likely to prefer Item 7 (n=3 and n=2, respectively). 


Table 13. Elementary school students’ Item preferences by participant group 


Student Demographic Category 

Preferred Item 
7 (Shapes 
Described) 

Preferred Item 
8 (Shapes not 
Described) 

No Preference 

LD-Reading (n=5) 

2 

2 

1 

ELL (n=4) 

3 

1 

0 

Low Vision (n=3) 

2 

1 

0 

TOTAL(n=12) 

7 

4 

1 


Students who preferred Item 7 stated that they liked how the computer voice read the shapes to 
them, which could help “a bit with getting the answer,” and because they did not “have to think 
about what the shape is.” One student who preferred Item 7 also mentioned that the slow speed 
of the read aloud can be helpful when items are difficult. Students who preferred Item 8 liked 
it because the read aloud finished faster, because they wanted to look at the shapes rather than 
have them described, or because they already knew their shapes and did not need the computer 
to tell them what each shape was. 

Issue 5: Do students prefer that answer options with numerais be read aioud? 

To answer whether students prefer that answer options with numerals be read aloud, an isomorph 
item set was developed. In Item Set 5, the first item read the numerals in the answer options 
aloud, while the second item did not read the numerals aloud. 

Replaying Audio 

For this item set, there were no students who replayed the read aloud after the item was initially 
heard. 

Reported Difficulty 

The reported difficulty level of Item Set 5 is shown in Tables 14 and 15. Most students who 
reported a difficulty level found these items to be Easy. 
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Table 14. Student reported difficulty of Item 9: numbers and answers read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

1 

3 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

6 

0 

1 

5 


Table 15. Student reported difficulty of Item 10: numbers and answers not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

1 

0 

1 

3 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

1 

0 

0 

2 

TOTAL(n=12) 

6 

0 

1 

5 


Item Preferences 


Table 16 shows students’ stated item preference by student category. Although the majority of 
students (n=8) preferred Item 9, where the numerals in the answer options were read aloud, 
there was some variation in preference by student need. Students with a reading-based learning 
disability were fairly evenly divided in their preference for the two items. For both ELLs and 
students with low vision, the majority preferred Item 9. 


Table 16. Elementary school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
9 (Numbers and 
Answers Read) 

Preferred Item 
10 (Numbers 
and Answers 
not Read) 

No Preference 

LD-Reading (n=5) 

2 

3 

0 

ELL (n=4) 

3 

1 

0 

Low Vision (n=3) 

3 

0 

0 

TOTAL(n=12) 

8 

4 

0 


All students who preferred Item 9 said that they liked having the computer read aloud the an- 
swer choice numbers. Some students mentioned that they were used to having the numbers read 
aloud on other tests, and others said it was easier for them to understand which option to choose 
because the voice had tied the number to the item answer choice. Students who preferred Item 
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10 either stated that they did not need the numbers read aloud, or that they liked the fact that 
the read aloud ended faster. 


Issue 6: Do students prefer that large numbers within items be read aioud? 

An item pair was developed in order to answer whether students prefer that large numbers 
within items be read aloud. In Item Set 6, for the first item, the computer read a large number 
(for example, “336”), while in the second item, the computer would simply say “this number,” 
when reading that part of the item. 

Replaying Audio 

The majority of students did not replay the audio in either of the items. One student with low 
vision replayed both Item 1 1 and 12. 

Reported Difficulty 

The reported difficulty level for Item Set 6 is shown in Tables 17 and 18. Fewer than half of the 
students reported a difficulty level for these items. Of those who did, they were closely divided 
between easy (n=3) and difficult (n=2). 


Table 17. Student reported difficulty of Item 1 1 : numbers read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

0 

0 

1 

4 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

0 

0 

1 

2 

TOTAL(n=12) 

3 

0 

2 

7 


Table 18. Student reported difficulty of Item 12: numbers not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=5) 

0 

0 

1 

4 

ELL (n=4) 

3 

0 

0 

1 

Low Vision (n=3) 

0 

0 

1 

2 

TOTAL(n=12) 

3 

0 

2 

7 


12 
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Item Preferences 


Students from all demographic categories showed a strong preference for Item 1 1 (n=10), where 
the numbers were read aloud within the item (Table 19). Only students with a reading-based 
learning disability were split on item preference, with three students preferring when the numbers 
were read aloud, and two preferring that the computer simply read “this number.” 


Table 19. Elementary school students’ Item preferences by participant group 


Student Demographic Category 

Preferred Item 
11 (Numbers 
Read) 

Preferred Item 
12 (Numbers 
not Read) 

No Preference 

LD-Reading (n=5) 

3 

2 

0 

ELL (n=4) 

4 

0 

0 

Low Vision (n=3) 

3 

0 

0 

TOTAL(n=12) 

10 

2 

0 


Students who preferred Item 1 1 stated that they liked that the computer read the numbers to 
them, and were frustrated that they had to read the numbers to themselves for Item 12. Some 
students commented that they “didn’t like [the second item] at all,” and that they could not rely 
on the read aloud while writing the problem down on scratch paper. Of the two students who 
preferred Item 12, one stated that the item felt “less stressful” when “this number” was read, 
rather than the actual number. The second student commented that they liked that the words 
were shaded during the read aloud of Item 12 (though word shading occurred for both items). 


Middle School Assessment With Audio Support 

The middle school assessment covered eight issues regarding how students preferred items be 
presented. The issues covered the following topics: 

• what level of description should be included in an item read aloud, 

• whether students want feedback for drag-and-drop items, 

• whether item equations should be read aloud, 

• whether diagrams should be described, 

• whether fractional ratios should be read aloud, 

• whether answer choice numerals should be read aloud, 

• large numbers within items should be read aloud, and 

• whether decimals should be read aloud using “decimal point” or place values (e.g., “five 
one-hundredths”). 
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Each issue is described in greater detail below, with student responses and preferences. A total 
of 17 students participated in the middle school assessment with audio support. Most students 
appeared to have answered the items earnestly. On a few occasions, students were observed to 
have guessed one or both items in a given item set. 

Issue 1 : Do students prefer that detailed information on iine graphs be read 
aioud or prefer that the graph oniy be described more generaiiy? 

To answer the question of whether students preferred that detailed information on line graphs be 
read aloud, an item pair was developed. In Item Set I, the first item read the points on the graph 
aloud to the student (e.g., in 1997, 50 million people used the Internet), while the second item 
provided a description of the graph title and axes, but did not read the graph points to the student. 

Replaying Audio 

There were no students who replayed the audio for either item. One student skipped the audio 
and answered both questions before the audio was finished reading, and another student skipped 
the audio on the second portion and answered the question before the read aloud was complete. 

Reported Difficulty 

Tables 20 and 21 show the reported difficulty level for Item Set I. All of the students who re- 
ported a difficulty level found the items in this set to be easy. 


Table 20. Student reported difficulty for Item 1 : detailed description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

3 

0 

0 

7 

ELL (n=3) 

2 

0 

0 

1 

Low Vision (n=4) 

1 

0 

0 

3 

TOTAL(n=17) 

6 

0 

0 

11 
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Table 21 . Student reported difficulty for Item 2: general description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

3 

0 

0 

7 

ELL (n=3) 

2 

0 

0 

1 

Low Vision (n=4) 

1 

0 

0 

3 

TOTAL(n=17) 

6 

0 

0 

11 


Item Preferences 

Table 22 shows which item students preferred across demographic categories. Regardless of 
their demographic category, students preferred Item 1, where the coordinate information on 
the grid was read aloud. Two students preferred Item 2, and two students from the LD-Reading 
group had no preference. 

Table 22. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
1 (Detailed 
Description) 

Preferred Item 2 
(General 
Description) 

No Preference 

LD-Reading (n=10) 

8 

0 

2 

ELL (n=3) 

2 

1 

0 

Low Vision (n=4) 

3 

1 

0 

TOTAL(n=17) 

13 

2 

2 


Students who preferred the first item stated that the read aloud was helpful because it read every 
part of the graph to them. One student with low vision commented that reading the points aloud 
was helpful because “otherwise my face is close to the screen and [I tend to] lose numbers.” 
Another student stated that when the computer read the numbers aloud, it made it easier for 
them to answer the question. One of the students who preferred the second item stated that the 
computer confused them when it read all of the graph points, while the other student disliked 
audio help in general, and preferred the second item because of this. The two students who did 
not have a preference did not provide additional information about their choice. 

Issue 2: Do students prefer hearing audio feedback for drag-and-drop items? 

For this issue, an item pair was developed that required students to drag and drop items onto 
a number line to show the correct placement of fractions. In Item Set 2, the first item did not 
provide feedback when students dragged items onto the number line. The second item told the 
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students that they had dropped the fractional equation onto the number line and where they had 
placed it. 


Replaying Audio 

There was one student in the LD-Reading group who replayed both Item 3 and Item 4, and one 
student with low vision who replayed Item 4 only. In addition, one student was frustrated by 
the read aloud, and told it to “shut up” during the second equation. 

Reported Difficulty 

Tables 23 and 24 highlight the reported difficulty level for Item Set 2. In this set, most of the 
students who reported a difficulty level found the items to be Difficult. 

Table 23. Student reported difficulty for Item 3: drag-and-drop no audio feedback 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

1 

1 

3 

5 

ELL (n=3) 

0 

0 

3 

0 

Low Vision (n=4) 

1 

0 

2 

1 

TOTAL(n=17) 

2 

1 

8 

6 


Table 24. Student Reported Difficulty for Item 4: Drag-and-drop Audio Feedback 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

0 

1 

3 

3 

ELL (n=3) 

0 

0 

3 

0 

Low Vision (n=4) 

1 

0 

2 

1 

TOTAL(n=17) 

1 

1 

8 

4 


Item Preferences 

The majority of students preferred Item 4, where the computer read the equation to them and let 
them know a box on the number line was filled each time they put an equation on the number 
line (Table 25). This trend held across all groups of students, with relatively few either express- 
ing a preference for Item 3, or not expressing a preference for either item. 
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Table 25. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
3 (Drag-and- 
Drop no Audio 
Feedback) 

Preferred Item 4 
(Drag-and-Drop 
Audio Feedback) 

No Preference 

LD-Reading (n=10) 

3 

4 

3 

ELL (n=3) 

0 

3 

0 

Low Vision (n=4) 

1 

3 

0 

TOTAL(n=17) 

3 

11 

3 


Students who preferred Item 3 stated that they liked how the item was read aloud. One student 
stated that it “gave you the info you need to know,” when working on the item. Students who 
preferred the second item (Item 4) said they liked that the computer read the equation when it 
was dropped onto the number line. One student mentioned that students who have low vision 
would know what equation they had moved, and where they had placed it on the number line. 
Two of the students who had no preference did not provide additional information and one stu- 
dent said that they liked parts of Item 3 and Item 4. This student mentioned that they thought 
it was annoying when the numbers were re-read, but liked that the read aloud confirmed which 
box the student had placed the equation into. 

Issue 3: Do students prefer that expressions and equations be read aioud? 

In Item Set 3, students were presented with two items that had algebraic equations. For Item 
5, the read aloud read the question and the equations. For Item 6, the read aloud read the ques- 
tion, but simply said “this expression” whenever an equation was highlighted. Students were 
required to determine which expressions in the answer choices were equal to the expression in 
the question. 

Replaying Audio 

There were three students, including one student with reading-related learning disabilities, one 
ELL, and one student with low vision replayed the audio for Item 5, where equations were read. 
One student with low vision began to complete the items before the read aloud had finished 
reading the problem to them. All other students waited until the read aloud was finished. 
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Reported Difficulty 


Tables 26 and 27 show the reported difficulty level for Item Set 6. Most of the students who 
reported a difficulty level stated that the items were difficult (n=7). A small number of students 
(n=3) reported the items to be easy. 


Table 26. Student reported difficulty for Item 5: equations read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

1 

1 

3 

5 

ELL (n=3) 

1 

0 

2 

0 

Low Vision (n=4) 

1 

0 

2 

1 

TOTAL(n=17) 

3 

1 

7 

6 


Table 27. Student reported difficulty for Item 6: equations not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

2 

1 

3 

4 

ELL (n=3) 

1 

0 

2 

0 

Low Vision (n=4) 

1 

0 

2 

1 

TOTAL(n=17) 

3 

1 

7 

6 


Item Preferences 


The majority of students preferred Item 5 (n=I3) where the equations were read aloud (see 
Table 28). Two students with reading-based learning disabilities preferred Item 5, and two had 
no preference. All students in the ELL and low vision groups preferred Item 5. 


Table 28. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
5 (Equations 
Read) 

Preferred Item 
6 (Equations 
not Read) 

No Preference 

LD-Reading (n=10) 

6 

2 

2 

ELL (n=3) 

3 

0 

0 

Low Vision (n=4) 

4 

0 

0 

TOTAL(n=17) 

13 

2 

2 
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Students who preferred Item 5 reported that they preferred hearing the equations read aloud 
to them. Some mentioned that when the equation was read aloud, they were able to recall it 
easier, while others mentioned that it helped them solve the equations. One of the two students 
preferred Item 6 because they disliked that the computer would say “open parentheses” and 
“end parentheses” when reading the equations, while the other student found the problems to 
be easy and did not think that having the equations read aloud was helpful. The two students 
who had no preference did not provide additional information about their choice. 

Issue 4: Do students prefer that diagrams be described? 

To discern whether students prefer that diagrams be described, an item pair (Item Set 4) was 
developed with layouts of a rectangular building with several rooms within. In Item 7, the com- 
puter read the title of the overall diagram, and then highlighted and read the name of each room. 
In Item 8, the title of the diagram was introduced, but individual rooms were not highlighted 
or read aloud. One student with a reading-based learning disability skipped the first item, and 
appeared to guess on the second. 

Replaying Audio 

Three students replayed the read aloud for this item set. One student with a reading-based learn- 
ing disability activated a portion of the read aloud by accident when trying to select an answer 
for Item 7. One student with a reading-based learning disability used the read aloud to have the 
answer choices re-read. In addition, one ELL had the computer re-read the diagram, but did not 
have it re-read any part of Item 8. 

Reported Difficulty 

The reported difficulty level for Item Set 4 is shown in Tables 29 and 30. Of the students who 
reported a difficulty level, most said these items were easy. 

Table 29. Student reported difficulty for Item 7: rooms read automatically 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

5 

0 

1 

4 

ELL (n=3) 

2 

1 

0 

0 

Low Vision (n=4) 

3 

1 

0 

0 

TOTAL(n=17) 

10 

2 

1 

4 
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Table 30. Student reported difficulty for Item 8: rooms not read automatically 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

4 

0 

0 

6 

ELL (n=3) 

2 

1 

0 

0 

Low Vision (n=4) 

3 

1 

0 

0 

TOTAL(n=17) 

9 

2 

0 

6 


Item Preferences 


As shown in Table 31, the majority of students preferred Item 7, where the rooms within the 
diagram were each read to the student by the computer. Two students from the LD-Reading 
group, and one student with low vision, preferred Item 8. A total of four students had no prefer- 
ence about how the items were presented. 


Table 31. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
7 (Rooms Read 
Automatically) 

Preferred Item 8 
(Rooms not Read 
Automatically) 

No Preference 

LD-Reading (n=10) 

4 

3 

3 

ELL (n=3) 

2 

0 

1 

Low Vision (n=4) 

3 

1 

0 

TOTAL(n=17) 

10 

3 

4 


The majority of students who preferred Item 7 mentioned that they liked that the computer read 
the names of each room. One student with a reading-based learning disability mentioned that 
students who were blind might not know which room was which without the read aloud. One 
of the students with low vision corroborated the student’s thought, mentioning that they would 
not know what the room labels were if the computer had not read them aloud. Students who 
preferred Item 8 stated that they liked that the read aloud finished faster, though some stated 
that the extended read aloud might be helpful for students who struggled with reading. Students 
with no preference thought the items were easy, and did not have a strong preference about how 
the items were presented. 

Issue 5: Do students prefer that fractional ratios be read aioud? 

To determine whether students prefer that fractions be read aloud, an item pair was developed. 
For items in Item Set 5, students were presented with a fraction and a list of answer options. 
They were required to select the answer option that was equal to the fraction in the problem. For 
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Item 9, the computer read the numbers in the numerator and denominator of the fraction and 
the numbers in the answer choices, while in Item 10, the computer simply said “this fraction 
is equal to...” and then presented the answer options without reading the numbers associated 
with that answer choice. One student with a reading-based learning disability skipped the first 
item and quickly answered the second item. 

Replaying Audio 

There were several students who replayed the audio during this item set. A total of six students 
replayed Item 9 and two of these students also replayed Item 10. Of these students, four students 
were in the LD-Reading group, one student had low vision, and one student was an ELL. 

Reported Difficulty 

Tables 32 and 33 show the reported difficulty level for Item Set 5. Most students did not report 
a difficulty level for this item set. Of those who did, there was no consensus on whether the 
items were easy, medium, or difficult. 


Table 32. Student reported difficulty for Item 9: equation read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

0 

0 

1 

9 

ELL (n=3) 

2 

1 

0 

0 

Low Vision (n=4) 

0 

0 

0 

4 

TOTAL(n=17) 

2 

1 

1 

13 


Table 33. Student reported difficulty for Item 10: equation not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

0 

0 

1 

9 

ELL (n=3) 

1 

1 

1 

0 

Low Vision (n=4) 

0 

0 

0 

4 

TOTAL(n=17) 

1 

1 

2 

13 
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Item Preferences 


As shown in Table 34, students clearly preferred Item 9, where the equation was read aloud by 
the computer. Although a few students preferred Item 10, the majority of students from each 
demographic category preferred to have the computer read the numbers in the fraction to them. 


Table 34. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 9 
(Equation Read) 

Preferred Item 
10 (Equation 
not Read) 

No Preference 

LD-Reading (n=10) 

8 

2 

0 

ELL (n=3) 

2 

1 

0 

Low Vision (n=4) 

3 

1 

0 

TOTAL(n=17) 

13 

4 

0 


Students who preferred Item 9 stated that they liked when the computer read the equation to them. 
Some students commented that this helped because they could hear the order of the numbers. 
One student with low vision mentioned that when the equation is not read aloud, some people 
would need to guess because the computer only reads the answer choices [and they cannot see 
the equation]. Students who preferred Item 10 said that they either found the equation read 
aloud to be confusing, or that they could read faster than the read aloud, and were annoyed that 
it did not keep up with them. 

Issue 6: Do students prefer that numerals in the answer options be read 
aioud? 

To answer whether students prefer numerals to be read aloud in item questions and answer 
choices, an item pair was developed. In Item Set 6, Item 1 1 read the actual numbers to students 
while reading a word problem. For Item 12, students were presented with a similar problem, but 
rather than reading the numbers, the computer read aloud “this number” and only read the answer 
choices (i.e.. A, B, C, D) to the student, rather than the numbers associated with each choice. 

Replaying Audio 

For this item set, two students with reading-based learning disabilities replayed the items. One 
student replayed both items, while the other replayed Item 12 (where the equation was not read 
aloud), and appeared to shake is head and chuckle when the computer read “this number” to 
the student. 
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Reported Difficulty 


Tables 35 and 36 show the reported difficulty level for Item Set 6. Most students did not report 
a difficulty level for this item set. No students reported these items to be difficult. Students were 
somewhat mixed on whether the items were easy or medium. 


Table 35. Student reported difficulty for Item 11 : numbers read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

2 

1 

0 

7 

ELL (n=3) 

0 

1 

0 

2 

Low Vision (n=4) 

2 

0 

0 

2 

TOTAL(n=17) 

4 

2 

0 

11 


Table 36. Student reported difficulty for Item 12: numbers not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

1 

1 

0 

8 

ELL (n=3) 

0 

1 

0 

2 

Low Vision (n=4) 

2 

0 

0 

2 

TOTAL(n=17) 

3 

2 

0 

12 


Item Preferences 


For this item set, students predominantly preferred Item II, where the numerals in the question 
and answer choices were read aloud by the computer. This trend held across all demographic 
groups (Table 37). 

Table 37. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
11 (Numbers 
Read) 

Preferred Item 
12 (Numbers 
not Read) 

No Preference 

LD-Reading (n=10) 

8 

1 

1 

ELL (n=3) 

3 

0 

0 

Low Vision (n=4) 

4 

0 

0 

TOTAL(n=17) 

15 

1 

1 
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Students preferred Item 1 1 because it read the numbers aloud to them. This allowed students to 
write down the numbers in the question and answer choices so that they could start solving the 
problem while the computer was completing the read aloud. Several students who preferred Item 
11 also mentioned that they disliked that the computer said “this number” for Item 12, rather 
than stating what the actual numbers were, with one student commenting that it is “unnatural 
not to have the numbers read aloud.” The two students who either preferred Item 12, or had no 
preference, did not provide any additional pertinent information about their choice. 

Issue 7: Do students prefer that large numbers within items be read aioud? 

To determine whether students prefer that large numbers be read aloud, students were given 
two items with large numbers (e.g., 327,577,529). Item 13 read each of the large numbers to the 
student, while Item 14 read the item questions, but did not read the numbers within the item to 
the student. Three students did not answer Item Set 7, bringing the total count of students for 
this item set from 17 to 14 students. 

Replaying Audio 

One student with a reading-based learning disability replayed Item 14, where large numbers 
were not read. 

Reported Difficulty 

The reported difficulty level for Item Set 7 is shown in Tables 38 and 39. Most students did not 
report a difficulty level for these items. Of those who did report a difficulty level, the students 
were fairly evenly divided in reporting the items to be easy and difficult. 


Table 38. Student reported difficulty for Item 13: large numbers read 


Student Demographic Category 


Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

0 

1 

0 

9 

ELL (n=3) 

1 

1 

0 

1 

Low Vision (n=4) 

1 

0 

2 

2 

TOTAL(n=17) 

2 

1 

2 

12 
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Table 39. Student reported difficulty for Item 14: large numbers not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

0 

1 

1 

8 

ELL (n=3) 

2 

0 

0 

1 

Low Vision (n=4) 

1 

0 

2 

1 

TOTAL(n=17) 

3 

1 

3 

10 


Item Preferences 


As shown in Table 40, students had a strong preference for Item 13, where the large numbers 
were read aloud. This trend held across all demographic groups. 


Table 40. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 13 
(Large Numbers 
Read) 

Preferred Item 14 
(Large Numbers 
not Read) 

No Preference 

LD-Reading (n=8) 

6 

2 

0 

ELL (n=2) 

2 

0 

0 

Low Vision (n=4) 

3 

0 

1 

TOTAL(n=14) 

12 

1 

1 


Students who preferred Item 13 stated that it was easier to follow the question because the com- 
puter read the large numbers aloud. One student mentioned that students with low vision might 
struggle to complete items where the numbers were not read aloud. One student mentioned that 
because the computer read the numbers aloud, they were able to start completing the problem 
in their head. One student with low vision stated that it was difficult to look up the numbers 
when the computer did not read them aloud. Neither student who preferred Item 14 or had no 
preference provided any additional information about their choice. 

Issue 8: Do students prefer that decimals be read aloud using singie digit 
names and the words “decimai point” or using their piace vaiue? 

Item Set 8 was developed to answer whether students prefer that items with decimals be read 
with “decimal point” or using their place value. In Item 15, the computer read the problem 
aloud and used “decimal point” to indicate a decimal was following (e.g., 8.95 was read “eight 
decimal point nine five”). Item 16 presented the item by giving the student the place value of 
the decimals (e.g., 8.95 was read “eight and ninety-five hundredths”). Four students did not 
complete Item Set 8, bringing the total number of students for this item set down to 13 students. 
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Replaying Audio 


One student with LD-Reading replayed the audio for Item 15, where the term “decimal point” 
was used. One student paused the audio before it had completed and answered the items without 
finishing the read aloud. 

Reported Difficulty 

Tables 41 and 42 highlight the reported difficulty level for Item Set 8. Most students did not 
report a difficulty level for these items. Of those who did, more students found the items to be 
easy. No students reported the items to be difficult. 


Table 41. Student reported difficulty for Item 15:decimals 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

1 

1 

0 

8 

ELL (n=3) 

1 

1 

0 

1 

Low Vision (n=4) 

2 

0 

0 

2 

TOTAL(n=17) 

4 

2 

0 

11 


Table 42. Student reported difficulty for Item 16: place value 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=10) 

2 

1 

0 

7 

ELL (n=3) 

0 

1 

0 

2 

Low Vision (n=4) 

2 

0 

0 

2 

TOTAL(n=17) 

4 

2 

0 

11 


Item Preferences 


As shown in Table 43, the majority of students preferred Item 15, where the numbers were read 
with “decimal point” rather than their place value. This was true across demographic groups, 
with only four students preferring Item 16. 
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Table 43. Middle school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
15 (Decimals) 

Preferred 
Item 16 (Place 
Value) 

No Preference 

LD-Reading (n=7) 

4 

3 

0 

ELL (n=2) 

2 

0 

0 

Low Vision (n=4) 

3 

1 

0 

TOTAL(n=13) 

9 

4 

0 


Students who preferred Item 15 stated that they found Item 16 to be confusing. Five students 
mentioned that they preferred to hear “point” rather than “decimal point,” because most of their 
teachers would present decimals using the former to indicate where the decimal point was placed. 
One student mentioned that “decimal point” was a long phrase to say. Students who preferred 
Item 16 either preferred it because they had gotten some practice with the first item, or could 
not provide additional information about their preference. Several students mentioned that they 
disliked Item 16 because the computer was perceived to mispronounce “millionths,” sounding 
closer to “smillionths,” which confused them. 


High School Assessment With Audio Support 

The high school assessment with audio support covered eight item sets. Similar to the elementary 
and middle school issues, the high school pairs of items covered a range of issues, including 
the following: 

1. how students prefer decimals, equations, and fractional ratios be read aloud, 

2. whether they wanted large numbers and audio feedback for drag-and-drop items read aloud, 

3 . whether students wanted detailed information on line graphs to be included in the read aloud. 

4. whether students wanted political cartoons described, and 

5. what level of detail students preferred when presented with information on maps. 

There were no high school students classified as ELLs who participated in the study, so this 
category is not included in the tables below. 

Most students were observed to have responded earnestly to the items at this grade school level. 
The researchers noted examples of students leaning in to read the question on the computer screen, 
asking for scratch paper, and making positive comments such as “I am totally loving this.” One 
student with visual impairment was observed to not respond earnestly to the items. Two students 
with learning disabilities were noted to not respond earnestly to some of the items. Students 
who did not respond earnestly made comments such as “Just guessed.” Also, students seemed 
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confused about how to answer, stating, “So does it mean you have to put words in the box or 
just numbers? I don’t know what to put in” and “How do I do it? I have no idea with this one.” 

Issue 1 : Do students prefer that decimals be read aloud using singie digit 
names and the words “decimai point” or using their piace vaiue? 

Item Set 1 was developed to answer whether students prefer that items with decimals be read 
with “decimal point” or using their place value. In Item 1, the computer read the problem aloud 
and used “decimal point” to indicate a decimal was following (e.g., 8.95 was read “eight decimal 
point nine five”). Item 2 presented the item by giving the student the place value of the decimals 
(e.g., 8.95 was read “eight and ninety-five hundredths”). 

Replaying Audio 

A total of three students replayed the audio in the items for this set. For Item 1, one student in 
the LD-Reading group and one student with low vision replayed the audio. The same student 
with low vision also replayed Item 2, and one additional student with low vision replayed Item 2. 

Reported Difficulty 

Tables 44 and 45 show the reported difficulty level for Item Set 1 . Fewer than half of the students 
reported a difficulty level for these items. Of those who did, more students (n=4) reported the 
items to be difficult. Two students reported the items to be easy. 


Table 44. Student reported difficulty for Item 1 : decimals 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

2 

0 

4 

4 

TOTAL(n=17) 

2 

0 

4 

11 


Table 45. Student reported difficulty for Item 2: place value 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

2 

0 

4 

4 

TOTAL(n=17) 

2 

0 

4 

11 
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Item Preferences 


Table 46 includes student preferences for the two items. Students who preferred the first item 
noted that they preferred decimals over fractions. One student commented that “people can get 
mixed up with fractions.” Students who preferred the second item commented that they find 
fractions easier than decimals. One student noted that the second item was “similar to how I was 
taught.” One student mentioned that for both items, the audio presentation was helpful because 
“I can hear it and also read it.” 

Table 46. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
1 (Decimals) 

Preferred Item 2 
(Place Value) 

No Preference 

LD-Reading (n=7) 

3 

3 

1 

Low Vision (n=10) 

8 

2 

0 

TOTAL(n=17) 

10 

5 

1 


Issue 2: Do students prefer that expressions and equations be read aioud? 

Item Set 2 addressed how expressions and equations were presented. In the first item, the num- 
bers and expressions were read aloud. In the second item in the pair, the equation was read as 
“equation.” 

Replaying Audio 

For Item Set 2, one student with a reading-related learning disability and one student with low 
vision replayed Item 3, where the equation was read. No students replayed the audio for Item 
4, where the equation was not read. 

Reported Difficulty 

Tables 47 and 48 show the reported difficulty level for Item Set 2. Most students did not report 
a difficulty level for these items. Of those who reported a difficulty level, students were divided 
between reporting the items as easy and reporting them as difficult. 
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Table 47. Student reported difficulty for Item 3: equations read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

1 

6 

Low Vision (n=10) 

3 

0 

2 

5 

TOTAL(n=17) 

3 

0 

3 

11 


Table 48. Student reported difficulty for Item 4: equations not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

2 

0 

3 

5 

TOTAL(n=17) 

2 

0 

3 

12 


Item Preferences 


Table 49 shows which of the two items in Set 3 the respondents preferred. As shown in the table, 
the majority of students (n=15) preferred that equations be read aloud. Only two students (one 
in each demographic category) stated that they did not like this option. 


Table 49. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 5 
(Equations Read) 

Preferred Item 6 
(Equations not 
Read) 

No Preference 

LD-Reading (n=7) 

6 

1 

0 

Low Vision (n=10) 

9 

1 

0 

TOTAL(n=17) 

15 

2 

0 


Issue 3: Do students prefer that fractional ratios be read aioud? 

Item Set 3 was developed to answer whether students prefer to have fractional ratios read aloud. 
In Item 5, the computer read the expression as a fraction, stating what was in the numerator and 
denominator. Item 6 was presented by reading the fractional ratio as “expression.” 
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Replaying Audio 


One student with a visual impairment and two students with visual impairments replayed the 
audio for Item 3. 

Reported Difficulty 

The reported difficulty level for Item Set 3 is given in Tables 50 and 51. Most students did not 
report a difficulty level for these items. Two students with low vision reported the items to be 
easy, and one student with low vision reported the items to be difficult. 

Table 50. Student reported difficulty for Item 5: fractional ratios read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

2 

0 

1 

7 

TOTAL(n=17) 

2 

0 

1 

14 


Table 51. Student reported difficulty for Item 6: fractional ratios not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

2 

0 

1 

7 

TOTAL(n=17) 

2 

0 

1 

14 


Item Preferences 

Table 52 shows high school students’ preferences for Item Set 3. In this Item Set, students 
overwhelmingly preferred to have the fractional ratios read aloud. Students commented that it 
was “more helpful” to hear the equations read aloud. 


NCEO 


31 





Table 52. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 5 
(Fractional 
Ratios Read) 

Preferred Item 6 
(Fractional 
Ratios not 
Read) 

No Preference 

LD-Reading (n=7) 

7 

0 

0 

Low Vision (n=10) 

8 

0 

2 

TOTAL (n=) 

15 

0 

2 


Issue 4: Do students prefer that large numbers within items be read aioud? 

Item Set 6 was developed to address the question of whether students preferred to have large 
numbers read aloud or not. In this item pair, all numbers in the first item were read aloud, includ- 
ing numbers in the answer choices. In the second item, all of the numbers in both the stimulus 
and answer choices are referred to as “this number.” 


Replaying Audio 

In this item set, no students replayed either test item. 

Reported Difficulty 

Only six students with low vision reported a difficulty level for the items in Item Set 4. Of these 
students, four found the items to be easy and two found the items to be medium. This was true 
for both items in the set. This information is reported in Tables 53 and 54. 

Table 53. Student reported difficulty for Item 7: large numbers read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

4 

2 

0 

4 

TOTAL(n=17) 

4 

2 

0 

11 
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Table 54. Student reported difficulty for Item 8: large numbers not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

4 

2 

0 

4 

TOTAL(n=17) 

4 

2 

0 

11 


Item Preferences 

High school student preferences for Item Set 4 are shown in Table 55. According to the table, 
all high school students preferred the first item, in which the large numbers were read aloud. 


Table 55. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred 
Item 7 (Large 
Numbers 
Read) 

Preferred 
Item 8 (Large 
Numbers not 
Read) 

No Preference 

LD-Reading (n=7) 

7 

0 

0 

Low Vision (n=10) 

10 

0 

0 

TOTAL(n=17) 

17 

0 

0 


When asked about why they preferred large numbers read aloud, one student noted, “It said ‘this 
number’ but I have no idea what the number is. People with no vision would not know what the 
number is.” Another student explained, “When not hearing [the number], I have to pay atten- 
tion and copy numbers off screen to scratch paper to work out an answer.” In addition, students 
expressed frustration when the equations were not read aloud. One student stated, “oh my gosh, 
just read the number.” Another explained, “My mind just naturally skips it, so I have to go back 
and re-read the expression.” Still another noted, “It is annoying how it just says ‘expression.’ It 
should say what the expression is.” 

Issue 5: Do students prefer hearing audio feedback for drag-and-drop items 
(feedback teiis the student what object was moved or what the drop area iooks 
iike with dragged objects)? 

In Item Set 5, students completed technology-enhanced items that required them to drag and 
drop equations into boxes that reflected an answer range. In the first item, no audio feedback 
was provided when the student initiated the drag-and-drop. In the second item, students received 
audio feedback related to the equation chosen as well as audio description of the box the equa- 
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tion was dragged into. The purpose of this item pair was to elicit student preferences for audio 
support in the drag-and-drop. 

Replaying Audio 

For this item set, one student in the LD-Reading demographic group replayed Item 9, where no 
audio feedback was provided. No students replayed Item 10. 

Reported Difficulty 

Only three students, two with LD-Reading and one with low vision, reported a difficulty level 
for Item Set 5. All three students reported the items to be difficult. This information is reported 
in Tables 56 and 57. 

Table 56. Student reported difficulty for Item 9: drag-and-drop not read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

2 

5 

Low Vision (n=10) 

0 

0 

1 

9 

TOTAL(n=17) 

0 

0 

3 

14 


Table 57. Student reported difficulty for Item 10: drag-and-drop read 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

1 

6 

Low Vision (n=10) 

0 

0 

1 

9 

TOTAL(n=17) 

0 

0 

2 

15 


Item Preferences 

High school students’ item preferences by demographic category are summarized in Table 58. 
The students in the LD-Reading group and Low Vision group are almost evenly divided in their 
item preferences, with one student with Low Vision expressing no preference for either item. 
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Table 58. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 9 
(Drag-and-drop 
not Read) 

Preferred 
Item 10 
(Drag-and- 
drop Read) 

No Preference 

LD-Reading (n=7) 

3 

4 

0 

Low Vision (n=10) 

3 

6 

1 

TOTAL(n=17) 

7 

9 

1 


One student who found the audio support to be helpful liked “that it actually tells you what you 
are dragging — you don’t want to grab the wrong thing.” Another student liked it for a similar 
reason, “because you know where you move it.” The student who did not like the drag-and-drop 
audio support noted that “it does not seem logical to hear where the item got moved because I 
moved it myself.” 

Issue 6: Do students prefer that detailed information on iine graphs be read 
aioud or prefer that the graph oniy be described more generaiiy? 

An item pair was developed to answer the question of whether students preferred to have detailed 
information on line graphs read aloud or if students preferred to have the information described 
more generally. In Item II, the graph is described in detail, including the titles for the x and y 
axis, and all of the points on the scatterplot are mentioned. In Item 12, only a general descrip- 
tion of the graph is provided. 

Replaying Audio 

No students replayed the audio for the items in this set. 

Reported Difficulty 

The reported difficulty level for Item Set 6 is shown in Tables 59 and 60. Most students did not 
report a difficulty level for these items. Of those who did, most students reported the items to 
be Difficult. One student reported the items to be Easy. 
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Table 59. Student reported difficulty for Item 11 : detailed description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

1 

4 

Low Vision (n=10) 

1 

0 

2 

7 

TOTAL(n=17) 

1 

0 

3 

11 


Table 60. Student reported difficulty for Item 12: general description 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

0 

7 

Low Vision (n=10) 

1 

0 

2 

7 

TOTAL(n=17) 

1 

0 

2 

14 


Item Preferences 


Table 61 summarizes the findings on students’ preferences for Item Set 6. The majority of stu- 
dents (n=10) preferred the line graph read aloud, although students with reading-related learning 
disabilities appear to be more split in their preferences. 


Table 61. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred Item 
11 (Detailed 
Description) 

Preferred Item 
12 (General 
Description) 

No Preference 

No Answer 

LD-Reading (n=7) 

4 

3 

0 

0 

Low Vision (n=10) 

6 

0 

1 

3 

TOTAL(n=17) 

10 

3 

1 

3 


One student noted that it is “easier” to have the detailed description, adding “people without 
vision could not read them so it is better to have it read.” One student who preferred the general 
description of the line graph said that the more detailed description “took too long.” 

Issue 7: Do students prefer that political cartoons be described? 

Item Set 7 was developed to address the question of whether or not students preferred to have 
political cartoons described. In Item 13, the political cartoon was described in detail. In Item 
14, the political cartoon was not described. 
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Replaying Audio 


One student with low vision replayed the audio for Item 13. No students replayed the audio 
for Item 14. 


Reported Difficulty 


Three students reported the items in Item Set 7 to be difficult. These results are reported in 
Tables 62 and 63. 


Table 62. Student reported difficulty for Item 13: cartoon described 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

2 

5 

Low Vision (n=10) 

0 

0 

1 

9 

TOTAL(n=17) 

0 

0 

3 

14 


Table 63. Student reported difficulty for Item 14: cartoon not described 


Student Demographic Category 

Easy 

Medium 

Difficult 

No 

Difficulty 

Reported 

LD-Reading (n=7) 

0 

0 

2 

5 

Low Vision (n=10) 

0 

0 

1 

9 

TOTAL(n=17) 

0 

0 

3 

14 


Item Preferences 

Table 64 shows how students responded to the question about which item they preferred in Item 
Set 7. Most students (n=9) preferred the cartoon to be orally described. It should be noted here 
that five students did not respond to this question. One of the respondents stated, “I liked when 
it described the cartoon. It made it easier to understand the question.” 


NCEO 


37 





Table 64. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred 
Item 13 
(Cartoon 
Described) 

Preferred 
Item 14 
(Cartoon 
not 

Described) 

No Preference 

No Answer 

LD-Reading (n=7) 

4 

2 

0 

1 

Low Vision (n=10) 

5 

1 

0 

4 

TOTAL(n=17) 

9 

3 

0 

5 


Several students who preferred to have the political cartoon described noted that it was much 
easier to tell what was happening in the cartoon with the added description. One student noted 
that “People with low vision could see what is in the picture this way.” Another student said, “I 
could tell what was happening.” 

Issue 8: Do students prefer that detailed information on maps be read aioud or 
prefer that the map oniy be described more generaiiy? 

Item Set 8 was developed to address the question of whether students preferred that detailed in- 
formation on maps be read aloud or whether students preferred to have the information described 
more generally. Item 15 included a longer reading passage that included a detailed description of 
a map. Item 16 was a longer reading passage that included only a general description of a map. 

Replaying Audio 

No students replayed either item in this item set. 

Reported Difficulty 

No students reported a difficulty level for Items 15 and 16. However, it should be noted that 
several students experienced testing fatigue and did not complete these items. Several students 
commented that they felt impatient listening to the voice read the longer narrative passages. 

Item Preferences 

High school students’ preferences for Item Set 8 are shown in Table 65. Most students with 
low vision preferred the map to be described with the audio support, while respondents with 
reading-related learning disabilities were more divided on this issue. 
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Table 65. High school students’ item preferences by participant group 


Student Demographic Category 

Preferred 
Item 15 
(Map 

Described) 

Preferred 
Item 16 
(Map not 
Described) 

No Preference 

No Answer 

LD-Reading (n=7) 

2 

3 

0 

2 

Low Vision (n=10) 

5 

1 

0 

4 

TOTAL(n=17) 

7 

4 

0 

6 


One of the students who liked the audio support stated the following, “I liked the one that read 
the map. It was better because it helps you visualize the picture in your head. It is better to have 
it read, especially if you have low vision.” And a student who preferred the second item pointed 
out, “It is better to see a map and not have it described.” 


Discussion 

The cognitive labs point to a number of findings related to the interactions of students with an 
assessment that has audio support. Overall, the findings from the study are mixed; the findings 
support the need to take an individual approach to assigning audio support as an accessibility 
feature or accommodation. 

Students typically preferred to have numbers read aloud. In several of the item sets, one item in 
the pair would have numbers read aloud, and in the other item, numbers would be read as “this 
number” or “equation.” In all of these cases, students in all three demographic groups preferred 
the item where the numbers were included in the audio support. This preference was strongest 
for students who had low vision. These students noted that it would be challenging to solve the 
problems without hearing the numbers read aloud. One student with in the LD-Reading group 
also noted that “some students might not be able to see the numbers, so how could they solve 
the problem?” 

Students also had a slight preference for numbers with decimals to be read as numerals with 
“decimal point” rather than to be read with place value. Students with reading-related learning 
disabilities were more evenly divided in both middle and high school than their peers with low 
vision. Students commented that it “makes more sense” to hear the numbers as numerals with 
“decimal point.” 

Students were mixed in their preferences for audio support with the drag-and-drop. For some 
students, this was a new item type. Other students were familiar with the drag-and-drop feature 
from computer games or instruction but had not encountered it in an assessment. A few students 
were familiar with this item type. Elementary school students generally preferred having the 
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audio feedback with the drag-and-drop items. Middle school students also preferred to have the 
audio feedback when they were using drag and drop. High school students were evenly split 
about their preference on this feature. It should be noted that some students who also used mag- 
nification with these items experienced some difficulties in navigating on the screen and reported 
frustration related to the audio support speaking when students would mouse on the screen. 

At the high school level, one item type addressed the description of political cartoons. Students 
generally preferred to have the cartoons described. Students commented that it would be easier 
for students with low vision to understand these items. Students in the LD-Reading group also 
preferred to have the cartoons described orally. 

Students generally reported that audio support helped them understand test questions better; 
however, this study did not examine the performance of students on the assessment. Most stu- 
dents responded earnestly to the test items, but some students commented that they knew the 
test was “just for fun” and spent less time trying to solve the problems. Other students reported 
difficulty with math and struggled to answer the test items. 

After the students completed the assessment, they were asked some additional questions about 
their preferences for how answer choices should be read aloud as well as their opinion of the 
computer-generated voice for the audio support. Students were asked if answer choices should 
be read as “Answer A,” “Answer Option A,” or simply as “A.” Table 66 shows a summary of 
student responses to this question. Three students did not respond to this question. Of those stu- 
dents who gave a preference, a majority of students (n=31) preferred to have the answer choice 
read as “Answer A.” Eight students preferred “Answer Option A,” and three students preferred 
simply “A.” One student stated a preference not to have the answer choices labeled at all. 


Table 66. Student preferences about how answer choices should be read 



Elementary 

School 

Middle 

School 

High School 

Total 

Answer A 

6 

14 

11 

31 

Answer Option A 

3 

1 

4 

8 

A 

1 

2 

0 

3 

Other (not reading 
answer iabeis) 

0 

0 

1 

1 

Total 

10 

17 

16 

43 


Students were also asked about their opinions of the computer voice used for the audio support 
in the cognitive labs. Generally, students thought the voice was acceptable. Some students com- 
mented that the voice sounded like a robot; others thought the quality was fine. One high school 
student noted, “The voice was decent — not too fast or slow. This voice is better than any other 
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voices I’ve heard using technology.” Students were mixed in their evaluation of the speed of 
the voice. Some thought it was too slow; others thought it was too fast. Several students com- 
mented that they prefer a human voice over a computer voice. Some students commented that 
they would prefer to have a male voice instead of a female voice or have two genders take turns. 

Limitations of the Study 

There were some limitations related to the audio support cognitive labs. These cognitive labs 
were conducted in three states. At the time of data collection, two of these states were members 
of the PARCC consortium and one state was not a member of a consortium. Efforts were made 
to include students from a state involved in Smarter Balanced, however, due to the demands of 
spring testing in general as well as consortium-related field testing, it was not possible to gather 
data from additional states. 

In addition, it was challenging to recruit ELLs to participate in the study. In many states, audio 
support is not a common accommodation for EEEs, and so it was difficult to secure district sup- 
port for including EEEs in the study sample. Many of the EEEs who participated in the study 
commented that they found that the audio support helped them understand the test questions 
better. This points to a need for further consideration of audio support as an accessibility tool 
or accommodation for EEEs. 

Another challenge in the study was that most of the items were math-related, and many students 
commented that they were not good at math. Some students reported that the test items appeared 
to be related to content they had not learned. This issue became somewhat of a limitation in two 
ways: 1) students were often more focused on the challenging content of the item than they were 
on how the audio support was helpful or not helpful, and 2) students sometimes began guessing 
at items and became less authentically engaged in the test itself. 

A final limitation related to the high school cognitive labs was that the longer reading passages 
did not appear to engage students, and many students reported the audio support to be frustrat- 
ing for the longer passages. Many students were less able to focus when the researchers got to 
these items and several students asked to skip these items. Thus, the sample for these items is 
much smaller than other items at the high school level. 

Conclusion 

The audio support cognitive labs administered as part of the Guidelines for Accessible Assess- 
ments Project were conducted with elementary, middle, and high school students with a range 
of individualized needs, including students who are EEEs, students who have low vision, and 
students who have a reading-related learning disability. The results of these cognitive labs 
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generally indicate that students prefer having more of the item content read aloud, including 
numbers in the stimulus and answer choices, as well as descriptions of pictures and graphics. 
However, in spite of these tendencies, the results were mixed overall. These findings support 
the general need to take an individualized approach to providing audio support for assessments. 
In developing online assessments, it is important to program audio support in ways that allow 
for multiple- approach (e.g., all parts of the item, a stimulus only, answer choices only, etc.) 
read aloud. 
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Appendix A: Study Instruments 


School #: 

Student #: 

Grade {circle one): 3 4 5 

Need {circle all that apply): Print Disability English Learner Low Vision 
Student Gender {circle one): Male Lemale 

Test Login ID: 

Note Taker: 

Date: 


Grades 3-5 Audio 

Thank you for participating in this study to understand more about how test questions are read 
aloud. We would like for you to view and answer 12 questions on the computer. Each question 
will be read to you, you can listen to all or parts of the item as many times as you like. We will 
ask you questions about what you are thinking as you answer the test questions. 

Item Set One: Coordinate Grids 

Observation Notes: 


After #I & #2: 

Was the way this graph was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this graph was read that was helpful? Can you 
please explain how the read aloud was helpful? 
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Was there anything/anything else about how this graph was read that was not helpful? Can you 
please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you just heard were presented differently. For the first question, the 
places of each house were read aloud. For the second question, places of each classroom were 
not read aloud. Which way do you think helped you answer the question? Did you like one way 
better than the other? If so, why did you like one of them better? 


Item Set Two: Drag and Drop 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, you were 
told when you dropped a soybean in the field and how many fields were planted. For the second 
question, you were not told anything when you dropped a com kernel on a field. Which way do 
you think helped you answer the question? Did you like one way better than the other? If so, 
why did you like one of them better? 
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Item Set Three: Equations 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 
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After the set: 


The two questions that you just heard were presented differently. For the first question, the 
equations were read aloud. In the second question, the equations were not read aloud. Which 
way do you think helped you answer the question? Did you like one way better than the other? 
If so, why did you like one of them better? 


Item Set Four: Diagram Description 
Observation Notes: 


After #I & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 
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Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you just heard were presented differently. For the first question, the three 
shapes were described. In the second question, the shapes were not described. Which way do 
you think helped you answer the question? Did you like one way better than the other? If so, 
why did you like one of them better? 


Item Set Five: Answer Options with Numerals 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 
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Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you just heard were presented differently. For the first question, the num- 
bers in the answer options were read aloud. In the second question, the numbers in the answer 
options were not read aloud. Which way do you think helped you answer the question? Did you 
like one way better than the other? If so, why did you like one of them better? 


Item Set Six: Large Numbers 
Observation Notes: 


After #I & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you just heard were presented differently. For the first question, the 
numbers were read. In the second question, “this number” was read instead of the numbers. 
Which way do you think helped you answer the question? Did you like one way better than the 
other? If so, why did you like one of them better? 
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After the Audio Test: 


When there were answer options, the read aloud said, “answer” before each answer option 
(like “Answer A, Answer B”). Was it clear what the answer options in each question were? Did 
you like having the word “answer” read aloud before the answer option, or would you prefer 
“Answer Option A, Answer Option B,” or just “A, B,” or nothing at all? 


For how many years have you received a read aloud accommodation during testing? 


Do you use a read aloud accommodation in the classroom (outside of testing)? Does someone 
read aloud to you at home? Do you use read aloud computer programs? 


Do you believe that using a read aloud accommodation helps you understand test questions better? 


What did you think about the voice of the read aloud we used today (quality, speed, etc.)? 
School #: 
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Student #: 

Grade {circle one): 6 7 8 

Need {circle all that apply): Print Disability English Learner Low Vision 
Student Gender {circle one): Male Lemale 

Test Login ID: 

Note Taker: 

Date: 


Grades 6-8 Audio 

Thank you for participating in this study to understand more about how test questions are read 
aloud. We would like for you to view and answer 16 questions on the computer. Each question 
will be read to you, you can listen to all or parts of the item as many times as you like. We will 
ask you questions about what you are thinking as you answer the test questions. 


Item Set One: Line Graphs 
Observation Notes: 


After #I & #2: 

Was the way this graph was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this graph was read that was helpful? Can you 
please explain how the read aloud was helpful? 
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Was there anything/anything else about how this graph was read that was not helpful? Can you 
please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you just heard were presented differently. For the first question, the 
locations of each point were read aloud. For the second question, the locations of each point 
were not read aloud. Which way did you think helped you answer the question? Did you like 
one way better than the other? If so, why did you like one of them better? 


Item Set Two: Drag and Drop 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, you were not 
told anything when you dropped an equation in a box. For the second question, you were told 
when you dropped an equation in a box on the number line, and whether a box was empty or 
filled. Which way did you think helped you answer the question? Did you like one way better 
than the other? If so, why did you like one of them better? 
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Item Set Three: Equations 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 
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After the set: 


The two questions that you heard were presented differently. For the first question, the equations 
were read aloud. In the second question, the equations were not read aloud. Which way did you 
think helped you answer the question? Did you like one way better than the other? If so, why 
did you like one of them better? 


Item Set Four: Diagram Description 
Observation Notes: 


After #I & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 
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Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, each room 
was highlighted as it was read aloud. In the second question, the room names were only read 
aloud when you clicked on them. Which way did you think helped you answer the question? 
Did you like one way better than the other? If so, why did you like one of them better? 


Item Set Five: Fractional Ratios 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


NCEO 


57 




Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the equation 
was read aloud. For the second question, the equation was read as “this equation.” Which way 
did you think helped you answer the question? Did you like one way better than the other? If 
so, why did you like one of them better? 


Item Set Six: Answer Options with Numerals 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the numbers 
in the answer options were read. In the second question, the numbers in the answer options 
were not read. Which way did you think helped you answer the question? Did you like one way 
better than the other? If so, why did you like one of them better? 
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Item Set Seven: Large Numbers 


Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 
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After the set: 


The two questions that you heard were presented differently. For the first question, the numbers 
in the question and in the answer options were read aloud. In the second question, numbers 
were not read aloud. Which way did you think helped you answer the question? Did you like 
one way better than the other? If so, why did you like one of them better? 


Item Set Eight: Decimals 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 
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Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the decimal 
numbers were read to you using their single digit names and the words “decimal point,” for 
example, “eight decimal point nine five.” In the second question, the numbers were read using 
their place value, for example, “eight and ninety five one hundredths.” Which way did you think 
helped you answer the question? Did you like one way better than the other? If so, why did 
you like one of them better? Would you prefer to hear just “point” instead of “decimal point”? 


After the Audio Test: 

When there were answer options, the read aloud said, “answer” before each answer option (like 
“Answer A, Answer B”). Was it clear what the answer options in each question were? Did you 
like having the word “answer” read before the answer option, or would you prefer “Answer 
Option A, Answer Option B,” or just “A, B,” or nothing at all? 


For how many years have you received a read aloud accommodation during testing? 


Do you use a read aloud accommodation in the classroom (outside of testing)? Does someone 
read aloud to you at home? Do you use read aloud computer programs? 
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Do you believe that using a read aloud accommodation helps you understand test questions better? 


What did you think about the voice of the read aloud we used today (quality, speed, etc.)? 
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School #: 

Student #: 

Grade {circle one): 9 10 11 12 

Need {circle all that apply): Print Disability English Learner Low Vision 
Student Gender {circle one): Male Lemale 

Test Login ID: 

Note Taker: 

Date: 

Grades 9-12 Audio 

Thank you for participating in this study to understand more about how test questions are read 
aloud. We would like for you to view and answer 16 questions on the computer. Each question 
will be read to you, you can listen to all or parts of the item as many times as you like. We will 
ask you questions about what you are thinking as you answer the test questions 

Item Set One: Decimals 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the decimal 
numbers were read using their single digit names and the words “decimal point,” for example, 
twenty-eight decimal point three. In the second question, the numbers were read using their 
place value, for example, twenty-three and three-tenths. Which way did you think helped you 
answer the question? Did you like one way better than the other? If so, why did you like one of 
them better? Would you prefer to hear just “point” instead of “decimal point”? 
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Item Set Two: Equations 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 
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After the set: 


The two questions that you heard were presented differently. For the first question, the equation 
was read aloud. In the second question, the equation was read as “this equation.” Which way 
did you think helped you answer the question? Did you like one way better than the other? If 
so, why did you like one of them better? 


Item Set Three: Fractional Ratios 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 
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Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the equation 
was read aloud. For the second question, the equation was read as “this equation.” Which way 
did you think helped you answer the question? Did you like one way better than the other? If 
so, why did you like one of them better? 


Item Set Four: Large Numbers 
Observation Notes: 


After #I & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 
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Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the numbers 
were read aloud. In the second question, “this number” was read instead of the numbers. Which 
way did you think helped you answer the question? Did you like one way better than the other? 
If so, why did you like one of them better? 


Item Set Five: Drag and Drop 
Observation Notes: 


After #I & #2: 

Was the way this question was read to you helpful? Why or why not? 
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If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, you were not 
told when you dropped an equation in a box. For the second question, you were told when you 
dropped an equation in a box, which box you dropped it into, and whether a box was empty or 
filled. Which way did you think helped you answer the question? Did you like one way better 
than the other? If so, why did you like one of them better? 
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Item Set Six: Line Graphs 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 
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After the set: 


The two questions that you heard were presented differently. For the first question, the locations 
of each point were read aloud. For the second question, the locations of each point were not 
read aloud. Which way did you think helped you answer the question? Did you like one way 
better than the other? If so, why did you like one of them better? 


Political Cartoons 
Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 


Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 
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Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, the picture 
was described. For the second question, the picture was not described. Which way did you 
think helped you answer the question? Did you like one way better than the other? If so, why 
did you like one of them better? 


Maps 

Observation Notes: 


After #1 & #2: 

Was the way this question was read to you helpful? Why or why not? 


If necessary, probe: 

Was there anything/anything else about how this question was read that was helpful? Can you 
please explain how the read aloud was helpful? 
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Was there anything/anything else about how this question was read that was not helpful? Can 
you please explain how the read aloud was not helpful? 


Was anything/anything else difficult to understand in the read aloud? Please explain what that was. 


Is there anything that you would change to make the read aloud better? Please explain what 
that would be. 


After the set: 

The two questions that you heard were presented differently. For the first question, each loca- 
tion on the map was read aloud. For the second question, a short general description was read 
aloud. Which way did you think helped you answer the question? Did you like one way better 
than the other? If so, why did you like one of them better? 


After the Audio Test: 

When there were answer options, the read aloud said, “answer” before each answer option (like 
“Answer A, Answer B”). Was it clear what the answer options in each question were? Did you 
like having the word “answer” read before the answer option, or would you prefer “Answer 
Option A, Answer Option B,” or just “A, B,” or nothing at all? 
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For how many years have you received a read aloud accommodation during testing? 


Do you use a read aloud accommodation in the classroom (outside of testing)? Does someone 
read aloud to you at home? Do you use read aloud computer programs? 


Do you believe that using a read aloud accommodation helps you understand test questions better? 


What did you think about the voice of the read aloud we used today (quality, speed, etc.)? 
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